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(54) Copyright protection for moving image data 

(57) In a system for playing back a moving image 
constructed by a set of objects, each object is efficiency 
ventiod and a problem of omission of a played back im- 
age due to delay time associated with verification 
processing is solved. On the transmission side, an MUX 
(201 ; generates an MPEG-4 bit stream obtained by cod- 
ing h plurality^ object and data streams constructing a 
moving image and IPMP (Intellectual Property Manage- 
ment and Projection) streams containing source infor- 
mation and use permission information into one stream. 
On the reception side, a DMUX (206) identifies and sep- 
arates each stream from the MPEG-4 bit stream and 
separates them into a plurality of object data and source 
inlormation/use permission information corresponding 



to each object data. On the basis of the separated 
source information/use permission information, an 
IPMP System (207) requests a corresponding server 
(202 - 204) to execute verification to access each object 
using a back-channel 1 (205) using an upstream or 
back-channel 2 (213) such as a telephone line. An ac- 
cess control point (212) is controlled to permit only ac- 
cess.to object data for which an access permission sig- 
nal has been received from the server. A signal for re- 
questing resending objects is transmitted to the distri- 
bution server (202 - 204) of objects for which verification 
is requested, using the back-channel 1 (205) or 2 (21 3), 
together with the source information of the request des- 
tination server and information representing the verifi- 
cation result. 
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Description 

BACKGROUND OF THE INVENTION 

5 FIELD OF THE INVENTION 

[0001] The present invention relates to a verification apparatus, verification method verification system, and storage 
medium and, more particularly, to a verification apparatus, verification method, verification system, and storage medi- 
um, which are suitable when verification is necessary for the purpose of copyright protection associated with individual 
10 objects of a moving image to be played back. 

DESCRIPTION OF THE RELATED ART 

[0002] Fig. 1 is a block diagram showing a conventional digital video data transmission/reception system. As shown 
*5 in Fig. 1 . a digital video data distribution server 10 downloads digital video data recorded in a digital video data storage 
device 12 such as a hard disk annexed to the distribution server 10 to a digital video data reception client 20 through 
a network 30 such as the Internet in response to a request from the reception client 20. The distribution server 10 has 
aconversion section 11 for coding digital video data. Digital video data is coded by the conversion section 11 to decrease 
the data amount, and distributed to the reception client 20 in accordance with a procedure such as the TCP/IP protocol. 
20 The reception client 20 has a conversion section 21 for decoding digital video data. A received digital video signal is 
played back by the conversion section 21 and displayed, recorded, or edited. 

[0003] An example of a system which constructs one moving image scene from a plurality of objects, codes and 
compresses each object by the conversion section 1 1 of the distribution server 10, transfers the objects to the reception 
client 20, and decodes and reconstructs the objects in the reception client 20 to play back the moving image scene is 
25 an MPEG-4 player. 

[0004] Fig. 2 is a block diagram of a conventional MPEG-4 player. Fig. 2 is based on "ISO/I EC FCD 14496-1 Fig. 
1 -1 \ and this has been described in detail in "ISO/IEC FCD 14496-1 ". Only a schematic arrangement will be described 
below. 

[0005] An MPEG-4 bit stream transmitted through a network or the like or MPEG-4 bit stream read out Irom a storage 
30 medium such as a DVD-RAM is received by TransMux Layer" in accordance with a procedure corresponding to trans- 
mission/read (session establishment) and separated into streams such as scene description information, object data, 
and object description data, decoded, and played back by a "FlexMux" section. On the basis of the scene description 
information, a scene is played back or graphically processed. 

[0006] Fig. 3 is a schematic and simple block diagram of the player shown in Fig. 2. When verification is necessary 
35 for the purpose of copyright protection for individual objects, a bit stream containing a plurality of object data including 
the scene description information may contain "IP Data Set" (copyright information group). 

[0007] However, even if the transmission bit stream contains "IP Data Set" (copyright information group), and "IP 
Data" is played back by "Object Descriptors" of the system shown in Fig. 2 or 3, "IP Data" is not processed in image 
playback processing. For this reason, "IP Protection" (copyright protection) processing is not executed. 
•*o [0008] In the system shown in Fig. 2 or 3, even when an "IP Data Set" (copyright information group) stream is con- 
tained in addition to the transmission bit stream, "IP Data" is not always played back by "Object Descriptors". Even if 
"IP Data" is played back by "Object Descriptors", "IP Data" is not processed in image playback processing. For this 
reason, "IP Protection" (copyright protection) processing is not executed. 

[0009] In this case, an application can receive decoded "IP Data Set" and execute 'IP Protection" processing. How- 
45 ever, this processing is unique to the application and is not always executed by another player or a player of another 
type. 

[0010] In the system shown in Fig. 2 or 3 : an image is played back after verification processing is performed for 
individual objects. For this reason, when new objects appear one after another in playing back a moving image scene, 
production must be temporarily stopped to require verification. 
50 [001 1] When verification is performed without stopping playback, images to be played back are omitted correspond- 
ing to the time required for verification. 

SUMMARY OF THE INVENTION 

55 [001 2] The present invention has been made in consideration of the above situation, and has as its object to efficiently 
execute verification processing, effectively protect a copyright or the like, effectively use copyright works : and solve 
the problem that a played back image is omitted due to the delay time associated with verification processing. 
[001 3] According to the first aspect of the present invention, there is provided a verification apparatus characterized 
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by comprising reception means for receiving moving image data containing use permission information from an external 
information processing apparatus, playback means for playing back part of the moving image data received by the 
reception means, input means for inputting characters/symbols, comparison means for comparing the use permission 
- information contained in the moving image data received by the reception means with the characters/symbols input 
5 by the input means, and transmission means for, when comparison by the comparison means has revealed that the 
use permission information matches the characters/symbols, transmitting an instruction signal for instructing to play 
back the moving image data again from first image data played back by the playback means to the external information 
processing apparatus. 

[0014] In the verification apparatus, the moving image data is preferably, e.g., compression-coded moving image 
io data. 

[0015] Preferably, the verification apparatus further comprises, e.g., storage means for storing the characters/sym- 
bols in advance, and the comparison means compares the use permission information contained in the moving image 
data received by 'the reception means with the characters/symbols stored in the storage means. 
[001 6] In the verification apparatus, the playback means preferably plays back, e.g. , time or spatial part of the moving 
15 image data. ' 

[001 7] In the verification apparatus, the transmission means preferably transmits, togetherwith the instruction signal, 
e.g., a signal representing that the verification apparatus can use the moving image data. 

[0018] In the verification apparatus, the transmission means preferably normally receives the moving image data 
using, e.g., an interface used to receive the moving image data in a normal direction and transmits the instruction signal 
20 using the interface in a reverse direction to. the normal direction. 

[0019] In the verification apparatus, the transmission means preferably transmits the instruction signal using, e.g.;. 
an upstream of an MP EG4 bit^stream: - • 

[0020] Preferably, the verification apparatus further comprises, e.g., an interface connected to a communication line, 
and the transmission means transmits the instruction signal through, e.g., the interface and communication line. ^ 

2S [0021] According to the second aspect of the.present invention,. there is provided a verification apparatus charac- 
terized by comprising reception means for receiving moving image data constructed by a plurality of object data in-d- 
eluding object data containing source information and usepermission information, separation means for separating- 
the moving image data received by the reception means into the plurality of object data, playback means for playing 
back ; of the object data separated by the separation means, object data which do not contain the use permission^ 

30 information, inputmeans for/inputting characters/symbols, comparison- means for comparing the use permission infor-^ 
mation with the characters/symbols input by the input means in association with, of the object data separated by the* 
separation means, the object data containing the use permission information, and transmission means for, when com-^ 
parison by the.comparison means hasrevealed that the>use, permission information.matches the.characters/symbols, 
transmitting an instruction signal for instructing to play back the moving image data again from first image data played 

35 back by the playback means to an external information processing.apparatus corresponding to the source information 
contained in the plurality of object data constructing the moving image data. 

[0022] In the verification apparatus, the source information is preferably, e.g., URL (Uniform Resource Locator) in- 
formation. 

[0023] According to the third aspect of the present invention, there is provided a verification method characterized 
40 by comprising the reception step of receiving moving image data containing use permission information from an external 
information processing method, the playback step of playing back part of the moving image data received in the re- 
ception step, the input step of inputting characters/symbols, the comparison . step of comparing the use permission 
information contained in the moving image data received in the reception step with the characters/symbols input in the 
input step, and the transmission step of, when comparison in the comparison step has revealed that the use permission 
45 information matches the characters/symbols, transmitting an instruction signal for instructing to play back the moving 
image data again from first image data played back in the playback step to the external information processing method. 
[0024] In the verification method, the moving image data is preferably, e.g., compression-coded moving image data. 
[0025] Preferably, the verification method further comprises, e.g., the storage step of storing the characters/symbols 
in advance, and the comparison step. comprises comparing, e.g., the use permission information contained in the 
so moving image data received in the reception step with the characters/symbols stored in the storage step. 

[0026] In the verification method, the playback step preferably comprises playing back, e.g., time or spatial part of 
the moving image data. 

[0027] In the verification method, the transmission step preferably comprises transmitting, together with the instruc- 
tion signal, e.g., a signal representing that the verification method can use the moving image data. 
55 [0028] In the verification method, the transmission step preferably comprises normally receiving the moving image 
data using, e.g., an interface used to receive the moving image data in a normal direction, and transmitting the instruc- 
tion signal using the interface in a reverse direction to the normal direction. 

[0029] In the verification method, the transmission step preferably comprises transmitting the instruction signal using, 
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e.g., an upstream of an MPEG4 bit stream. 

[0030] Preferably, the verification method further comprises using, e.g., an interface connected to a communication 
line, and the transmission step comprises transmitting the instruction signal through, e.g., the interface and communi- 
cation line. 

5 [0031] According to the fourth aspect of the present invention, there is provided a verification method characterized 
by comprising the reception step of receiving moving image data constructed by a plurality of object data including 
object data containing source information and use permission information, the separation step of separating the moving 
image data received in the reception step into the plurality of object data, the playback step of playing back, of the 
object data separated in the separation step, object data which do not contain the use permission information, the input 

io step of inputting characters/symbols, the comparison step of comparing the use permission information with the char- 
acters/symbols input in the input step in association with, of the object data separated in the separation step, the object 
data containing the use permission information, and the transmission step of, when comparison in the comparison step 
has revealed that the use permission information matches the characters/symbols, transmitting an instruction signal 
for instructing to play back the moving image data again from first image data played back in the playback step to an 

75 external information processing method corresponding to the source information contained in the plurality of object 
data constructing the moving image data. 

[0032] In the verification method, the source information is preferably, e.g., URL (Uniform Resource Locator) infor- 
mation. 

[0033] According to the fifth aspect of the. present invention, there is provided a verification system having an infor- 
20 malion processing apparatus and a verification apparatus, which are connected to communicate with each other, char- 
acterized in that the verification apparatus comprises reception means for receiving moving image data containing use 
permission information from the information processing apparatus, playback means for playing back part of the moving 
image data received by the reception means, input means for inputting characters/symbols, comparison means for 
comparing the use permission information contained in the moving image data received by the reception means with 
2S the characters/symbols input by the input means, and transmission means for, when comparison by the comparison 
moans has revealed that the use permission information matches the characters/symbols, transmitting an instruction 
signal tor instructing to play back the moving image data again from first image data played back by the playback 
moans to the information processing apparatus. 

[0034] According to the sixth aspect of the present invention, there is provided a verification system having at least 
jo one information processing apparatus and a verification apparatus connected to the information processing apparatus 
to communicate with each other, characterized in that the verification apparatus comprises reception means for re- 
ceiving moving image data constructed by a plurality of object data including object data containing source information 
and use permission information from the information processing apparatus, separation means for separating the moving 
image data received by the reception means into the plurality of object data, playback means for playing back, of the 
35 object data separated by the separation means, object data which do not contain the use permission information, input 
means lor inputting characters/symbols, comparison means for comparing the use permission information with the 
characters/symbols input by the input means in association with, of the object data separated by the separation means, 
the ooject data containing the use permission information, and transmission means for, when comparison by the com- 
parison means has revealed that the use permission information matches the characters/symbols, transmitting an 
40 instruction signal for instructing to play back the moving image data again from first image data played back by the 
playback means to the information processing apparatus corresponding to the source information contained in the 
plurality ol object data constructing the moving image data. 

[0035] According to the seventh aspect of the present invention, there is provided a storage medium which stores a 
program that can be executed by an apparatus, characterized in that the apparatus for executing the program is caused 

45 to operate as an apparatus comprising reception means for receiving moving image data containing use permission 
information from an external information processing apparatus, playback means for playing back part of the moving 
image data received by the reception means, input means for inputting characters/symbols, comparison means for 
comparing the use permission information contained in the moving image data received by the reception means with 
the characters/symbols input by the input means, and transmission means for, when comparison by the comparison 

50 means has revealed that the use permission information matches the characters/symbols, transmitting an instruction 
signal for instructing to play back the moving image data again from first image data played back by the playback 
means to the external information processing apparatus. 

[0036] According to the eighth aspect of the present invention, there is provided a storage medium which stores a 
program that can be executed by an apparatus, characterized in that the apparatus for executing the program is caused 
55 to operate as an apparatus comprising reception means for receiving moving image data constructed by a plurality of 
object data including object data containing source information and use permission information, separation means for 
separating the moving image data received by the reception means into the plurality of object data, playback means 
for playing back, of the object data separated by the separation means, object data which do not contain the use 
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permission information, input means for inputting characters/symbols, comparison means for comparing the use per- 
mission information with the characters/symbols input by the input means in association with, of the object data sep- 
arated by the separation means, the object data containing the use permission information, and transmission means 
for, when comparison by the comparison means has revealed that the use permission information matches the char- 

s acters/symbols, transmitting an instruction signal for instructing to play back the moving image data again from first 
image data played back by the playback means to an external information processing apparatus corresponding to the 
source information contained in the plurality of object data constructing the moving image data. 
[0037] According to the ninth aspect of the present invention, there is provided a verification apparatus characterized 
by comprising separation means for separating image data constructed by a plurality of object data containing source 

10 information into the plurality of object data and source information of the plurality of object data, management means 
for managing the plurality of source information separated by the separation means, transmission means for transmit- 
ting a verification signal to an information device on a network, which is specified by arbitrary source information man- 
aged by the management means, reception means for receiving a permission signal transmitted from the information 
device in response to the verification signal transmitted by the transmission means, and control means for enabling 

15 use of object data containing the arbitrary .source information on the basis of the permission signal received.by the 
reception means; 

[0038] In the verification apparatus, the source information is preferably, e.g., URL (Uniform Resource Locator) in- 
formation. 

[0039] In the verification apparatus, the image data constructed by the plurality of object data is preferably, e.g., 
20 compression-coded moving image data. - 

[0040] The verification, apparatus preferably further comprises, e.g., selection means for selecting arbitrary source^ 
information managed by the management means. -■ 

[0041] In the verification apparatus, the transmission means preferably normally receives the image data using, e. 
g., an interface used to receive the image data constructed by the plurality of object data in a normal direction and; ^ 

25 transmits the verification signal using the interface in a reverse direction to the normal direction. ^mm? 
[0042] In the verification apparatus, the transmission means preferably transmits the verification signal using, e.g., 
an upstream of . an MPEG4 bit stream. -r 
[0043] Preferably, the verification apparatus further comprises an interface connected to a communication line, the ^ 
transmission means transmits the verification signal to the information device through the interface and communication - 

30 line, and the reception means receives the permission signal from the information device through the communication 
line and interface. 

[0044] According to the 10th aspect of the present invention, there is provided a verification method characterized;,- 
by comprising the separation step of separating Jmage data constructed* by.a plurality of .object data containing source 
information into the plurality of object data and source information of the plurality of object data, the management step.- - 

35 of managing the plurality of source information separated in the separation step, the transmission step of transmitting 
a verification signal to an information device on a network, which is specified by arbitrary, source information managed 
in the management step, the reception step of receiving a permission signal transmitted from the information device 
in response to the verification signal transmitted in the transmission step, and the control step of enabling, use of object 
data containing the arbitrary source information on the basis of the permission signal received in the reception step. 

40 [0045] According to the 1 1 th aspect of the present invention, there is provided a verification system having an infor- 
mation device and a verification apparatus, which are connected to a network, characterized in that the verification 
apparatus comprises separation means for separating image data constructed by a plurality of object data containing 
source information into the plurality of object data and source information of the plurality of object data, management 
means for managing the plurality of source information separated by the separation means, transmission means for 

45 transmitting a verification.signat to the information device on a network, which is specified by arbitrary source information 
managed by the management means, reception means for receiving a permission signal transmitted from the infor- 
mation device in response to the verification signal transmitted by the transmission means, and control means for 
enabling use of object data containing the arbitrary source information on the basis of the permission signal received 
by the reception means. 

so [0046] According to the 12th aspect of the. present invention, there is provided a storage medium which stores a 
program for controlling verification processing, characterized in that an apparatus for loading and executing the program 
is caused to operate as an apparatus comprising separation means for separating image data constructed by a plurality 
of object data containing source information into the plurality of object data and source information of the plurality of 
object data, management means for managing the plurality of source information separated by the separation means, 

55 transmission means for transmitting a verification signal to an information device on a network, which is specified by 
arbitrary source information managed by the management means, reception means for receiving a permission signal 
transmitted from the information device in response to the verification signal transmitted by the transmission means, 
and control means for enabling use of object data containing the arbitrary source information on the basis of the per- 
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mission signal received by the reception means. 

[0047] According to the 1 3th aspect of the present invention, there is provided a verification apparatus characterized 
by comprising coding means for coding streams of object data constructing image data and streams of source infor- 
mation of the object data into one stream, generation means for generating a decoded stream from the plurality of 

5 streams coded by the coding means, separation means for separating the decoded stream generated by the generation 
means into the streams of the plurality of object data contained in the decoded stream and the streams of the source 
information of the plurality of object data, management means for managing the source information of the object data, 
and transmission means for transmitting the plurality of streams of source information separated by the separation 
means to the management means. 

10 [0048] In the verification apparatus, the source information is preferably URL (Uniform Resource Locator) informa- 
tion. 

[0049] In the verification apparatus, the image data constructed by the plurality of object data is preferably compres- 
sion-coded moving image data. 

[0050] Preferably, the verification apparatus further comprises, e.g., designation means for designating a type of 
15 each stream coded by the coding means, and also identification means for identifying the type of each stream separated 
by the separation means, and sending means for sending each stream to an appropriate decoder on the basis of the 
type identified by the identification means. 

[0051] According to the 14th aspect of the present invention, there is provided a verification method characterized 
by comprising the coding step of coding streams of object data constructing image data and streams of source infor- 

20 mation of the object data into one stream, the generation step of generating a decoded stream from the plurality of 
streams coded in the coding step, the separation step of separating the decoded stream generated in the generation 
step into the streams of the plurality of object data contained in the decoded stream and the streams of the source 
information of the plurality of object data, the management step of managing the source information of the object data, 
and the transmission step of transmitting the plurality of streams of source information separated in the separation 

25 step to the management step. 

[0052] In the verification method, the source information is preferably URL (Uniform Resource Locator) information. 
[0053] In the verification apparatus, the image data constructed by the plurality of object data is preferably compres- 
sion-coded moving image data. 

[0054] The verification method preferably further comprises, e.g., the designation step of designating a type of each 
30 stream coded in the coding step, and also the identification step of identifying the type of each stream separated in 
the separation step, and the sending step of sending each stream to an appropriate decoder on the basis of the type 
identified in the identification step. 

[0055] According to the 15th aspect of the present invention, there is provided a verification system comprising a 
transmission apparatus and a reception apparatus, characterized in that the transmission apparatus comprises coding 

35 means for coding streams of object data constructing image data and streams of source information of the object data 
into one stream, generation means for generating a decoded stream from the plurality of streams coded by the coding 
means, and transmission means for transmitting the decoded stream generated by the generation means to the re- 
ception apparatus, and the reception apparatus comprises separation means for separating the decoded stream trans- 
mitted from the transmission apparatus into the streams of the plurality of object data contained in the decoded stream 

40 and the streams of the source information of the plurality of object data, management means for managing the source 
information of the object data, and sending means for sending the plurality of streams of source information separated 
by the separation means to the management means. 

[0056] According to the 16th aspect of the present invention, there is provided a storage medium which stores a 
program that can be executed by an apparatus, characterized in that the apparatus for executing the program is caused 

45 to operate as an apparatus comprising coding means for coding streams of object data constructing image data and 
streams of source information of the object data into one stream, generation means for generating a decoded stream 
from the plurality of streams coded by the coding means, separation means for separating the decoded stream gen- 
erated by the generation means into the streams of the plurality of object data contained in the decoded stream and 
the streams of the source information of the plurality of object data, management means for managing the source 

50 information of the object data, and transmission means for transmitting the plurality of streams of source information 
separated by the separation means to the management means. 

[0057] Other features and advantages of the present invention will be apparent from the following description taken 
in conjunction with the accompanying drawings, in which like reference characters designate the same or similar parts 
throughout the figures thereof. 
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BRIEF DESCRIPTION OF THE DRAWINGS 
[0058] 

5 Fig. 1 is a block diagram showing a conventional digital video data transmission/reception system; 

Fig. 2 is a block diagram showing the arrangement of a conventional MPEG-4 player; 
Fig. 3 is a block diagram showing the schematic and simple arrangement of the player shown in Fig. 2; 
Fig. 4 is a block diagram showing the arrangement of an MPEG-4 player according to a preferred embodiment of 
the present invention; 

10 Fig. 5 is simple- block diagram for explaining remote access; 

Fig. 6 is a view showing an example of a hierarchical structure when a URL destination further has a URL desig- 
nation; 

Fig. 7 is a flow chart showing client operation associated with verification processing; 

Fig. 8 is a block diagram showing the arrangement shown in Fig. 3, to which an IPMP System processing section 
is is further added: . 

Fig. 9 is a view showing the internal functional block diagram and data flow of an MPEG-4 player; 
Fig. 10 is a view simply showing a data processing process shown in Fig. 5; 

Fig. 11 is a flow chart showing an example of time adjustment operation of an MPEG-4 object access data unit; 
Fig. 12 is a view showing data movement and timing of a Decoding Buffer and Composition Memory; 
20 Fig. 1 3 is a view showing a dataprocessing process in the arrangement shown in Fig. 6 : to which an IPMP System 

processing section- is added; and 

Fig. 14 is a flow chart showing an operation exampleof the IPMP System shown in Fig. 8. 
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

25 

[0059] Preferred embodiments of the present invention will be described below with reference to the accompanying 
drawings. The following embodiments are related to a system for efficiently executing verification processing using a 
so-called "back-channel". 

30 (First Embodiment) 

[0060] Fig. 4 is a block diagram showing the schematic arrangement of a system including an MPEG-4 player ac- 
cording to a preferred embodiment of the present invention. The system shown in Fig. 4 operates "IP Data" to realize 
"IP Protection". The system shown in Fig. 4 has an IPMPS (Intellectual Property Management and Protection System): 
35 207 and is different from the system shown in Fig. 3 in that the copyright verification and protection function are realized 
by this IPMPS 207. 

[0061] Fig. 7 is a flow chart showing client operation associated with verification processing. Operation of the system 
shown in Fig. 4 will be described below with reference to Fig. 7. On the server side, a multiplexer 201 receive individual 
objects from a plurality of network sites 202 to 204 having different URLs (Uniform Resource Locators) such as URL1 , 

40 URL2, and URL3, and generates moving image data constructed by the plurality of objects. This moving image data 
is transmitted to a client through the network in response to a request from the client as an MPEG-4 bit stream 205. 
[0062] In step S1 , the client receives the MPEG-4 bit stream 205 from the server. Each object constructing the MPEG- 
4 bit stream has information (URL information) representing the copyright holder. In step S2, the client separates the 
received MPEG-4 bit stream into a plurality of streams such as the plurality of objects and accompanying information 

45 (including URL information) by a demultiplexer 206. The URL information of each object is sent to the IPMPS 207 as 
part of "IPMP Stream" as a stream of "IP Data". 

[0063] In step S3, one URL information is selected from one or a plurality of pieces of URL information sent to the 
IPMPS 207. To do this, the operator may designate the information, or the IPMPS 207 may select the information in 
accordance with a predetermined order. 
so [0064] In step S4, a verification request signal is transmitted to the server 201 with a corresponding URL in one or 
a plurality of servers connected to the network on the basis of the selected URL information. In this case, a back- 
channel 1 or back-channel 2 to be described later is used for transmission. 

[0065] In stop S5, the client waits for an access permission signal transmitted from the server 201 that has received 
the verification request signal. When the access permission signal is received, the flow advances to step S6. When 
55 no access permission signal is received in a predetermined time, the flow advances to step S7. 

[0066] In step S6, when the access permission signal is received, an object for which access permission (verification) 
has been obtained can be accessed. More specifically, by enabling a control signal 21 2 for controlling an access control 
point, a corresponding stream (i.e., the stream of an object for which access is permitted by the access permission 
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signal) from the demultiplexer 206 can be accessed by a scene descriptor 208, audio visual decoder 209, and object 
descriptor 210. 

[0067] In step S7, by disabling the control signal 21 2 tor controlling an access control point, access to a corresponding 
stream (i.e., the stream of an object for which access permission has not been obtained regardless of the verification 
s request) from the demultiplexer 206 by the scene descriptor 208, audio visual decoder 209 : and object descriptor 210 
is inhibited. 

[0068] In step S8, it is confirmed whether URL information added to another object is present. If YES in step S7, the 
flow returns to step S3; otherwise, the series of processing operations is ended. 

[0069] A scene/graphic processing section 211 performs scene synthesis and graphic processing on the basis of 
10 data supplied from the scene descriptor 208, audio visual decoder 209, and object descriptor 210. Only objects for 

which access permission has been obtained may be synthesized for playback. Alternatively, when an object for which 

access permission has not been obtained is present, playback may not be performed at all. 

[0070] The above-described verification processing will be described below in more detail. 

[0071] An MPEG-4 stream contains "ES_Descriptor" that describes the contents of "Elementary Stream" (ES) as a 
f 5 bit stream in units of objects and "OD_Descriptor° that describes the object itself. When URL information for designating 

a command and an access destination for remote access is obtained in "ES_Descriptor" or "OD_Descriptor", remote 

access is executed in accordance with a procedure shown in Fig. 5. 

[0072] Fig. 5 is a schematic block diagram for explaining remote access. Referring to Fig. 5, "DAI" is an interface 
layer between an MPEG-4 bit stream and network, which is called "DMIF Application Interface*. Details have been 

20 described in the "ISO/IEC 14496-6 DMIF document DMIF Application Interface" and will be omitted herein. 

[0073] The MPEG-4 bit stream also contains "DecoderConfigDescriptor" representing information of a decoder type 
corresponding to "Elementary Stream" (ES). This "DecoderConfigDescriptor 0 is a structure formed. from some data 
elements. One of the elements is a 1 -bit upStream parameter representing the stream type. Details have been described 
in the "ISO/IEC 14496-1 FCD 8. 3. 4. DecoderConfigDescriptor" and will be omitted herein. 

25 [0074] Expression 1 is an example of "DecoderConfigDescriptor". 

(Expression 1 : DecoderConfigDescriptor) 
30 aligned (8) class DecoderConfigDescriptor 

: bit (8) tag=DecoderConf igDescrTag { 
bit (8) length; 

35 

bit (8) objectProf ilelndication; 
bit (6) streamType; 
40 bit(l) upstream; 

const bit(l) reserved=l; 
bit (24) bufferSizeDB; 
bit (32) maxBitrate; 
bit ( 32 ) avgBitrate; 

DecoderSpecif iclnfo decSpecif iclnfo [ ] ; 



45 



SO 



ss [0075] The stream is identified on the basis of the value of "streamType" as a data element in the class declaration 
of "DecoderConfigDescriptor" of expression 1. The value of "streamType" is defined as in Table 1 . 
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Table 1 



Stream Type Designation Value 


Stream Type Designation Value 


Stream Type 


0x00 


reserved for ISO use 


0x01 


Ob j ect D esc ri pto r St r earn 


0x02 


ClockReferenceStream 


0x03 


SceneDescriptionStream 


0x04 


VisualStream 


0x05 


AudioStream j 


0x06 


MPEG7Stream 


0x07-0x09 


reserved for ISO use 


OxOA. 


ObjectContentlnfoStream 


OxOB 


IPMPStream ; 


0x0C-0x1F 


reserved for ISO use 


Ox20-0x3F 


user private 



[0076] In Tnble 1 , a value for identifying "IPMP Stream" unique to this embodiment is added to the "ISO/lEC 14496-1 
25 FCD Tiblc 0-1 sireamType Values". Parameters or terms in Table 1 are the same as in the M ISO/IEC 14496-1 FCD'V 
and n dctmied description thereof will be omitted. 

[0077] As described above, in Table 1 , the value for identifying "IPMP Stream" unique to this embodiment is added. 
Thio "iPMr Stream - is originally contained in a source code for the multiplexer 201 for generating the MPEG-4 bit 
sir aim 

30 [0078] in this source code for the multiplexer 201 , "IPMP Stream" is defined by expression 2 (*) below. 
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[Expression 2: Mux source] 
{ 

objectDescriptorlD 0 
es_descriptor [ 
( 

es__N umber 1 
fileName Inline. od 
streamType BIFS 
streamPriority 5 
decConf igDescr { 

streamType 2//0D Stream 

bufferSizeDB 200 

} 

alConf igDescr { 

useAccessUnitStartFlag TRUE 
useAccessUnitEndFlag TRUE 
useRandomAccessPointFlag TRUE 
useTimeStampsFlag TRUE 
timeStampResolution 1000 
timeStampLength 14 

} 

} 
{ 

es_Number 2 
streamType BIFS 
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streamPriority 5 
f ileName In] ine . bif 
decConf igDescr { 

streamType 4//BIFS Stream 

bufferSizeDB 1000 

) 

alConf igDescr { 

useAccessUnitStartFlag TRUE 
useAccessUnitEndFlag TRUE 
useRandomAccessPointFlag TRUE 
useTimeStampsFlag TRUE 
tirneStampResolution 100 
timeStampLength 14 
0CR_ES_Id 1 

} 
} 

] 

objectDescriptorlD 33 
esjdescriptor [ 
{ 

es_N umber 1 
f ileName t2 
streamType G723 
streamPriority 4 
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decConf igDescr { 

streamType 6 //AudioStream 

prof ileAndLevelIndication0xcl//G723 

bufferSizeDB 300 

) 

alConf igDescr { 

. timeStampResolution 1000 
compositionUnitRate 30 

} 

extensionDescriptor I PMP_Descriptor Pointer { 
IPMP_Descriptor_ID 69 

» 

es__N umber 2 
fileName tl 
streamType H263 
decConf igDescr { 

streamType 8 //IPMPStream (*) 

bufferSizeDB. 1600 

» 

alConf igDescr { 

useAccessUnitStartFlag TRUE 
useAccessUnitEndFlag TRUE 
useRandomAccessPointFlag TRUE 
useTimeStampsFlag TRUE 
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timeStampResolution 1000 
timeStampLength 10 
P DU_s e qN umLengt h 3 
AU_seqNumLength 8 
0CR_ES_Id 2113 

} 

} 

] 

objectDescriptorlD 32 
es_descriptor { 
es_Number 1 
fileName tl 
streamType H263 
decConf igDescr { 

streamType 5//VisualStream . 

profileAndLevelIndication0xC2//H263 

bufferSizeDB 1600 

} 

alConf igDescr { 

useAccessUnitStartFlag TRUE 
useAccessUnitEndFlag TRUE 
useRandomAccessPointFlag TRUE 
useTimeStampsFlag TRUE 
timeStampResolution 1000 
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timeStampLength 10 

5 . PDU_seqNumLength 3 

AU_seqNumLength 8 
OCR_ES_Id 2113 

10 

) 

} 



[0079] In expression 2, when "objectDescriptorlD" is "33", "IPMP Stream" is defined. This means that a portion rep- 
resenting the stream of an object protected by "IPMP Stream" is contained in "objectDescriptorlD 33". 
[0080] The stream type of "IPMP Stream" is defined as "streamTypeS". This meaning is the same as that of "0x0b" 
20 defined as the stream type designation value of "IPMP Stream" in Table 1 . 

[0081] In this embodiment, the multiplexer 201 collects such source codes and adds "IPMP Stream" to the stream 
of a plurality of objects, thereby generating an MPEG-4 bit stream coded into one stream. 

[0082] When the stream is identified by the above-described " Decode rConfigDescriptor 1 ', not only "IPMP Stream" 

but also the streams of objects can be separated from one stream. 
2S [0083] As shown in Fig. 4, when "DecoderConfigDescriptor. upStream" as a flag representing the direction of a 

stream is set at "1 ", the system is set in the "upstream" state to transmit a stream from the client side to the server 

side. In this case, a transmission function using the state of the "upstream" state will be referred to as " back-channel 1 ". 

[0084] In normal playback, "DecoderConfigDescriptor. upStream" is at "0", so a "downstream" state in which a stream 

is transmitted from the server side to the client side is set. When permission for access to an object is wanted, "De- 
30 coderConfigDescriptor. upStream" is set at "1", and so-called "back-channel 1" for "upstreaming" necessary data to 

the URL destination is used to send "IPMP Management Data" (copyright management information) to the server side 

as "IPMP Stream", so response data is transmitted from the URL destination by remote access. 

[0085] "IPMP Stream" shown in Table 1 has "IPMP ES" and M IPMP_D n . Each "IPMP.ES" is formed from a series of 

"IPMP_Messages". Expression 3 is an example of description of "IPMP_Messages". 

35 

[Expression 3: IPMP__Message] 
class IPMP_Message () { 

40 

unsigned int(8) IPMPS_TypeCount ; 
bit (1) hasURL; 
45 int i; 

for (i = 0; i < IPMPSJTypeCount ; i++) { 
unsigned int (16) IPMPJType [ [i ] ] ; 

50 



55 
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unsigned int(32) offset [[i]]; 
5 unsigned int(16) length[[i]]; 

I 

if (hasURL) { 

10 

unsigned int(5) lengthOf URLbits; 
bit (3) reserved=Oblll; 
75 unsigned int (lengthOf URLbits) lengthOf URL; 

char ( 8 ) URLString [ lengthOf URL] ; 

} 

20 

for (i = 0; i < IPMPSJTypeCount; i++) { 
char ( 8 ) I PMP_data [ length [ i ] ] ; 

25 ) 

} - 

[0086] Inexpression 3, "IPMPSJypeCount" represents the number of different "IPMPtypes". Since different IPMPSs 
30 can exist, "IPMP_Messages" can correspond to a plurality of IPMPSs. 

[0087] When a URL is designated, "IPMPS_TypeCount" has a value "0". Otherwise, "IPMPS_TypeCount" has "1" * 
as a minimum value. In this case, "IPMP_Message" stored in an external device is referred to and used in place of 
internariPMP. Message 0 . 

[0088] "IPMPS_D B is formed from "IPMP Descriptor". This "IPMP Descriptor- is a data structure for performing spe- 
35 cific IPMP control for each "elementary streams". "I PMP Descriptor Updates" is executed as part of an object descriptor 
stream. Expression 4 is an example of description of "IPMP Descriptor Updates". 

[Expression 4: !PMP_J3escriptorUpdate] 

40 

aligned (8) class IPMP_DescriptorUpdate: unit (8) 
IPMPJDescriptorUpdateTag { 
45 unsigned int (8) descriptorCount ; 

int i; 

for (i =0; i < descriptorCount; i++) { 

so 

I PMP_Descriptor d [ [ i ] ] ; 

) 



[0089] In expression 4, "descriptorCount" represents the number of "IPMP_Descriptors D to be updated, and d[i] rep- 
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resents a certain "IPMPJDescriptor". 

[0090] Expression 5 is an example of description of IPMPJDescriptor". 

[Expression 5: IPMPJDescriptor] 
class IPMP_Descriptor () { 

bit (8) IPMP_Descriptor_ID; 

unsigned int(8) IPMPSJTypeCount ; 

bit(l) hasURL; 

int i; 

for (i = 0; i < IPMPSJTypeCount; i++) { 

unsigned int (16) IPMPSJType [ [i] ] ; 
unsigned int (32) offset [[i]]; 
unsigned int (16) length([i]]; 



} 

if (hasURL) { 

unsigned int (5) lengthOf URLbits ; 
bit ( 3 ) reserved=0bl 1 1 ; 

unsigned int (lengthOf URLbits) lengthOf URL; 
char (8) URLString [lengthOf URL] ; 

} 

for (i = 0; i < IPMPSJTypeCount; i++) { 
char (8) IPMP_data [length [i] ] ; 

} 

} 

[0091] In expression 5, n IPMP_DescriptorJD D is a number unique to each "IPMPJDescriptor". "ESJDescriptors" 
refer to 'IPMP_Descriptors" using the "IPMP_DescriptorJD". "IPMPSJTypeCount" represents the number of different 
IPMPSs designated by "IPMP^message". 

[0092] Fig. 6 is a view showing an example of a hierarchical structure when a URL destination further has a URL 
designation. Fig. 6 shows a two-layer structure. If still another URL designation is present, a three- or four-layer structure 
may be formed. Referring to Fig. 6, although "IPMPStream" is not clearly illustrated, "IPMP_ES° or "IPMPJD" associ- 
ated with an object to be remote-designated is decoded and remote-accessed in correspondence with "SceneDescrip- 
tionStream" or "ObjectDescriptionStream 0 , as needed, as in the above-described Fig. 5. 

[0093] Verification processing using the "upstream" state of the MPEG-4 bit stream, i.e., using the back-channel 1 
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has been described above. This verification processing using the "back-channel 1" is "upstream" processing in real- 
time bit stream playback and therefore assumes high-speed processing with a relatively small data amount and short 
processing time. In a real-type playback system delay due to remote access and verification using the "back-channel 
1 " is preferably as small as possible. 
5 [0094] However, even when the data amount is small, verification may require a considerable time. This poses a 
problem of delay in the "back-channel 1 Fromthe viewpoint of the allowable delay time?and necessity of interactive 
operability, a second "back-channel" is preferably prepared. 

[0095] For this purpose, in this embodiment, an I/O (interdevice Input/Output) interface different from that for trans- 
mitting the MPEG-4 bit stream is used. This will be called a "back-channel 2". 
w [0096] Before a description of verification processing using the "back-channel 2", the relationship between the data 
amount and delay time in the "back-channel 1" and "back-channel 2" will be considered. In the report of "MPEG-4 
Requirement Group", the allowable delay time of "back-channel 1" that does not impede real-time playback is 1 frame 
time. On the basis of this, the relationships between the assumed data amount and bit rate in the "back-channel 1° 
and "back-channel 2" are shown in Table 2. 

15 - 

Table 2 



Delay Times and Data Amounts of back-channels 1 and 2 


Notation 


Use Purpose 


Data Amount 


Delay Time 


back-channel 1 


high-speed IPMP remote access for verification 


3000 - 5000 bits/s ' 


100 - 300 ms 


back-channel 2 


low-speed IPMP input/output access for verification 




> 500 ms 



[0097] In high-speed IPMP remote access for verification, the delay time is limited in processing a data amount within 
2£ 100 to 500 bits/frame through a transmission line with a bit rate of 3K to 5K/sec. The relationship between, . 
"IPMPJvlessage" data or n IPMP_Description" data and delay-bandwidth as a result of "remote content access" by 
"back-channel" can be regarded as Table 2, so the data amount for actual verification is limited. Verification often 
requires time asynchronously with stream processing. 

[0098] Verification of a.plurality of objects may be executed not in one site but in a plurality of sites. In this case, the 
3 0 conditions in Table 2 become stricter and are not suitable for practical use. Hence, for a verification procedure which 
allows low-speed processing asynchronously with stream processing, the "back-channel 2° is preferably used. 
[0099] Processing using the "back-channel 2" will be described below. The "back-channel 2" for low-speed IPMP 
input/output access for verification is used as an I/O (interdevice Input/Output) interlace different from that for trans- 
mitting the MPEG-4 bit stream, as shown in Fig. 4. 
35 [0100] A computer terminal 214 having a keyboard, display, and modem is prepared next to the "back-channel 2" 
and connected to the telephone line and IPMPS 207. In this arrangement, the computer terminal 214 receives an object 
in a stream, which requiresverification, and information of the verification destination from the IPMPS 207 and displays 
the information on the display. The operator selects an object in the stream, which requires verification, by referring to 
the display. The computer terminal 214 calls the verification destination, receives the verification method or access 
40 code from the verification destination, and displays the contents on the display. When the operator inputs the received 
information using the keyboard, the IPMPS 207 is notified of the input information and enables access to the necessary 
object. 

[01 01] Use of a telephone line has been exemplified above. Instead, a cable of CATV or radio communication channel 
may be used. 

45 [0102] Alternatively, a PC card storing information necessary for access verification, which has been acquired by a 
contract with the verification destination in advance, is inserted into the PCMCIA interface in the computer terminal 
214 to notify the IPMPS 207 of information necessary for verification and enable access to the object, as needed. 
[0103] For verification processing for which the operation time or verification time becomes relatively long, this meth- 
od is effective for processing other than real-time processing, e.g., at the time of starting stream playback or scene 

so change. 

[0104] As described above, according to the first embodiment, the "back-channel 1" or "back-channel 2" can be 
selected and used in accordance with the application purpose. This selection may be performed by the operator, or 
the optimum back-channel may be selected in the system in consideration of the delay time limit or the like. 
[0105] When two different "back-channels" are prepared, flexible verification processing can be realized. 

ss 

(Second Embodiment) 

[0106] As described above, in the first embodiment, an IPMP Stream containing URL information is added to the 
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streams of a plurality ot objects, and these streams are coded into one stream to generate an MPEG-4 bit stream. In 
addition, a stream containing IPMP Stream is identified from this MPEG bit stream, and a "back-channel" is used when 
the MPEG-4 player transmits the verification request signal to a server having a corresponding URL in one or a plurality 
of servers connected on the network. In the second embodiment, another method of using the "back-channel 0 will be 
5 described. 

[0107] Fig. 8 is a block diagram showing the schematic arrangement of an MPEG-4 player having the arrangement 
shown in Figs. 2 and 3, to which a copyright protection system (IPMP System 86) and object data processing flow 
control section (IPMP Stream Flow Control 83) are added. 

[0108] Fig. 8 shows the contents of stream control at the "access control point" in Fig. 4 in more detail. 
io [0109] Referring to Fig. 8, an MPEG-4 bit stream containing coded image object data requiring copyright protection 
is divided into object data by a Demux Layer 81 and converted/synchronized to the time in the player by a Sync Layer 
82 in accordance with time stamp information added in coding or bit stream generation. 

[0110] On the other hand, the IPMP System 86 performs verification processing for object data requiring copyright 
protection, which are separated into individual data, on the basis of copyright protection information separated by the 

is Demux Layer 81 and transmits a permission signal to the IPMP Stream Flow Control 83 to perform object data process- 
ing flow control. In a Compression Layer 84, each object data is decoded by a decoder in units of object data. In a 
Composition Layer 85, a scene is synthesized in accordance with the decoded scene description, and displayed. 
[0111] Especially, there are some object data processing flow control methods. In this embodiment, the problem to 
be solved will be described by exemplifying Test Conditions #1 and #2. 

20 [01 1 2] Table 3 shows four test plans as examples of the relationship between the IPMP System (IPMPS) and Stream 
Flow Control. 



25 



Table 3 IPMP Test Plan 



Test 1 



Test 2 



Test 3 



Test 4 



30 



35 



40 



t Sl(Ca) S2(Cv) 



MPEG-4 



t Sl(Ca) S2(Cv) 



MPEG-4 



IPMPS1 



SI'' (SI <Ca)) 



t Sl(Ca) S2(Cv) 



MPEG-4 



IPMPS2 



S2~ l (S2(Cv)) 



t SJ(Ca) S2(Cv) 



MPEG-4 



IPMPS1 



Sl~* (SI (Ca)) 



IPMPS2 



S2' 2 (S2(Cv)) 



45 



[0113] In Table 3, test 1 shows a case wherein no IPMP systems are present; test 2, a case wherein only the IPMPS1 
is present; lest 3, a case wherein only IPMPS2 is present; and test 4, a case wherein both the IPMPS1 and IPMPS2 
are present. 

so [0114] Input/output signals in the respective tests and the difference in role between the IPMPS1 and IPMPS2 will 
be described next. 

[0115] In Table 3, Unprotected Text Object Stream is expressed by 1", Protected Audio Stream is expressed by "S1 
(Ca)*\ and Protected Video Stream is expressed by B S2(Cv) n . 

[0116] S1 (Ca) IPMP System is expressed by "IPMPS1 and the XOR result (logical exclusive OR) between original 
ss coded data and ASCII code V is expressed by "SlfCa)". hence, the interpretation key is ASCII code °x fl , and the 
output is the n XOR° between the original coded data and V. 

[0117] S2(Cv) IPMP System is expressed by "IPMPS2 0 , and the XOR result between original coded data and ASCII 
code "a" is expressed by °S2(Cv) tt . hence, the interpretation key is ASCII code n a B , and the output is the "XOR" between 



EP 0 969 668 A2 



the original coded data and "a". 

[0118] "Graceful Error" means an error on the output side of the decoder, which occurs when the protected object 
stream cannot be normally interpreted by the key. "Graceful Error" that may occur in, e.g., a Protected Video Stream 
is error such as "no display" or "display of distorted image". In only test 4, no "Graceful Error" occurs. 
[0119] Table 4 shows the.conditions and parameters of IPMP Verification test. 



Table 4 

IPMP Verification Test Condition and Parameters 



I Test 1 " Test 2 [ Test 3 | Test A "| 



Condition 
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c 


a 


Unprotected 


<— 


<— 


<— 


o 




Text 








n 
1 1 


ol (La ) 




<— 


<— 


t— 
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Audio 
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S2 (Cv) 


r roceccea 


<— 


<— 




n 




Video 








»- 

L 
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I 


IPMP-ES and ipmp-d 


yes 


yes 


yes 


yes 
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o 












n 












a 
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ir laenLiiicaLion 


yes 


yes 


yes 


yes 
















IPMP-S1 


none 


XOR x tor 
bi iua ) 


none 


XOR x for 

£>1 (Uaj 






none 


none 


vnp "a" fr\r 

AUrv a i or 


auk a ror 
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81 


none 


Embedded 


<- 


<- 


e 






key & 






s 






constant 






t 






delay 






C 












o 












n 












d 












i 












t 












i 












o 












n 
















none 


User 
interaction & 
non- fixed 
delay 








Synchronization 


yes 


yes 


yes 


yes 




Expected result 


a ;pass 


a ;pass 


a ;pass 


a ;pass 






SI (Ca) ; error 


SI (Ca) ;pass 


SI (Ca) ;error 


SI (Ca) ;pass 






S2 (Cv) /error 


S2 (Cv) ; error 


S2{Cv);pass 


S2(Cv) ;pass 



[0120] In Table 4, when test 2 is to be executed, under Test Condition #1, a normal key for each object stream is 
55 present in the IPMP system (IPMPS1 and IPMPS2) in advance so that an incoming object stream is immediately (or 
with a predetermined delay time) "interpreted 0 and output to each decoder. 

[0121] When test 2 is to be executed, under Test Condition #2, a normal key for each object stream is not present 
in the IPMP system (IPMPS1 and IPMPS2) in advance. The normal key is input from external keys or by a user inter- 



a 
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active method such as smart card insertion, and an incoming object stream is "interpreted" and output to each decoder 
For this reason, the delay time is not constant. 

[01 22] Fig. 9 shows the internal functional block diagram and data flow of an MPEG-4 player. 
[0123] Fig. 9 shows a simple arrangement of an actual system for a description of a synchronization mechanism, 
s and the IPMP system and object data processing flow are not illustrated. 

[0124] First, an entry function ExecuteQ of the MPEG-4 System Player started from an application starts functional 
modules, ensures a data area buffer, allocates the memory to each functional function, and prepares for data process- 
ing. 

[0125] MPEG-4 bit streams input by a FlexDemux 91 as a Service module function of the DMIF layer, i.e., packet 
io data or data files from the network are received as a series of data groups-and transferred to an ALManager 92. 

[0126] In the ALManager 92, object data such as video data, audio data, and scene description information are 
separated from the data group. The scene description information or object-associated information data are transferred 
to a BIFS Decoder 93, and video and audio data are transferred to a Decoder 94 as data channels. 
[0127] In accordance with the scene description information decoded by the BIFS Decoder 93 and Decoder 94 and 
'5 time stamp information added in bit stream generation, a Presenter 95 or Media Stream data processing section (not 
shown) adjusts the time relationship between the decoded Media Object data (Video and Audio data), synchronizes 
them, and synthesizes a scene. * 
[0128] Fig. TO simply shows the above series of data processing processes. 

[0129] Referring to Fig. 10 : the FlexDemux 91 receives an MPEG-4 bit stream and separates it into elementary 
20 streams (ES) in units of object data. The ALManager 92 divides the-ES of each object data in units of decoding units. 
The BIFS Decoder 93 and Decoder 94 decode each object data. A data group Media ^Stream of the decoded object 
data is generated. The Presenter 95 executes time adjustment between the individual object data using the "MediaS- 
treamimp retch ()■ function for processing Media Stream data, synthesizes the object data into one scene, and dis- 
plays the sccne : • . . 
zs [0130] Fig n is a flow chart showing a data processing example of time adjustment. Time adjustment processing 
by tho Prosonior 95 will be described in detail with reference to Fig. 11 . 

[0131] In step S1101 , an allowance value is added to the current time of the System Player (-^dwCurrentTime). On 
the dhsis o f this value, the stamp time (TimeStamp) of data to be processed (AU) is converted into the System Player 
limo m slop S1102 MdwTime). In step S1104, the current time (dwCurrentTime) is compared with the stamp time 

30 (dwTimo of the data to be processed (AU). When the stamp time (dwTime) of the data to be processed (AU) is later 
than the current lime (dwCurrentTime), the flow advances to step S1 106* to synthesize an actual scene. If the stamp 
lime ;dwTmc) of the data to be processed (AU) is earlier than the current time (dwCurrentTime) /it is determined that 
the data is unsuitable for scene synthesis .(it is-determined that the data ts not in good time for scene synthesis), and 
the flow advances to step S1105 to process the next data processing block (AU). 

35 [0132) Fig 12 is a timing chart showing time adjustment processing shown in Fig. 11 in time series. 

[0133] Referring .to Fig. 12, an Object stream (AUO) arrives at a Decoding Buffer 1201 of the BIFS Decoder 93 or 
Decode- 94 at lime Arrival (AU0), is decoded, and sent toa Composition Memory 1 202 of the Presenter 95 at stamp 
. time DT3 ( AUO) added upon encoding. A scene is synthesized from scene synthesis'time CTS (CU0). The next Object 
stream (AU1 ) is also transferred from the Decoding Buffer 1201 to the Composition Memory 1202 at time DTS (AU1), 

40 and a scone is synthesized from time CTS (CU1 ). 

[0134] As is apparent from Fig. 12, in Fig. 11, the time DTS in the Decoding Buffer 1201 is adjusted to the actual 
scene synthesis time CTS in the Composition Memory 1 202,. that is later than the actual current time dwCurrentTime. 
[0135] In Fig 1 3, processing in the IPMP system is added to the processing flow shown in Fig. 10. More specifically, 
the following processing is performed. 

-is [0136] Ihc process in which the FlexDemux 91 receives an MPEG-4 bit stream and separates it into elementary 
streams (ES) m units or object data, and the ALManager 92 divides the ES of each object data in units of decoding 
units is the same as in Fig. 10. Next, a protected stream is specified from the object data divided by the ALManager 
92. especially on the basis of IPMP-associated information, and IPMP System processing such as normal key input 
and vesication is performed. The BIFS Decoder 93 and Decoder 94 decode the Media Stream as a data group to be 

so decoded in units of object data. The Presenter 95 adjusts time of each object, synthesizes a scene, and displays it. 
[0137] Object data processing flow control under Test Conditions #1 and #2 in executing test 2 shown in Table 4 will 
be described below. First, under Test Condition #1 , the key interpretation time is transmitted to the decoder as a pre- 
determined delay in units of IPMP Systems. For this reason, when the entire delay is set to be within the range where 
it can bo absorbed by the Compression Layer 84 in Fig. 8 or Presenter 95 in Fig. 9, no problem of synchronization occurs. 

55 [0138] Under Test Condition #2, the following processing is performed. 

[0139] Fig 14 is a flow chart for explaining processing of the IPMP system in executing test 2 under Test Condition #2. 
[0140] In step S1401 . the stream of each object divided by the ALManager 92 is obtained. In step S1402, it is de- 
termined whether valid key input is present. If NO in step S1 402, the flow advances to step S1 403 to HOLD processing 
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without interpreting the protected stream. If YES in step S1402, the flow advances to step S1404 to interpret the 
protected stream. Then, the flow advances to the next processing. 

[0141] When flow control shown in Fig. 14 is performed in executing test 2 under Test Condition #2, streams until 
the normal key input are suspended. On the other hand, non-protected streams or streams which have already been 
s verified and interpreted by normal key input are transferred to the subsequent time synchronization processing for 
decoder processing and scene synthesis. The elapse time until the previous suspended streams are verified and 
interpreted by normal key input and transferred to the next processing is not constant because of the user interactive 
operation for each protected stream. In addition, at the processing resumption time, the dwTime may already pass the 
dwCurrentTime. 

10 [0142] In this case, as is apparent from Figs. 11 and 12, the streams for which processing has been resumed are 
not decoded until the dwTime after resumption becomes later than the dwCurrentTime. Processing skips to the next 
data to be processed (AU) (i.e., the data is thinned). Skipped portions are not synthesized into a scene. 
[01 43] As described above, under Test Condition #2, the data is partially thinned, so continuous contents cannot be 
obtained from the first time. 

is [0144] In "pusrT-type data distribution such as pay TV, one-directional data distribution is basically performed in 
accordance with the time zone, and data is received by an image reception system with verification function, e.g., a 
set top box. Since this system can be sufficiently coped with by Test Condition #1, no problem is posed. 
[0145] However, for example, assume that a viewer sees a commercial demonstration contents group of the first 
several minutes of, e.g., a movie and selects one content. If he/she will acquire and enjoy the video data after charging 

20 and verification, this case cannot be coped with by Text Condition #1 . Under Test Condition #2, since playback is 
resumed after selection/verification, the viewer cannot obtain some contents, that have already been broadcasted. 
[0146] The MPEG-4 allows selection/playback in units of video objects. For this reason, in the above commercial 
demonstration contents, even when verification processing is not performed, some objects such as persons or back- 
ground can be kept played back as protected streams and graceful error. In this case as well, since playback is resumed 

25 after selection/verification under Test Condition #2, the viewer cannot obtain normal and full contents that have already 
been broadcasted. 

[0147] When the viewer will enjoy full contents from the beginning, he/she must instruct the server on the contents 
distribution side to resend the video data from the beginning. 

[0148] As a general solution, the client (user) side requests the server (contents distribution) side to resend the full 

30 contents at the time of resuming video playback after selection/verification. Normally, to issue this request, the server 
side need provide an application to the client side in advance to receive the request from the client side. 
[0149] However, when a scene is to be synthesized by obtaining a plurality of video object contents or audio object 
contents from different URL destinations (Uniform Resource Locator), as in the MPEG-4, applications and verification/ 
resending methods for the plurality of contents distribution servers are necessary. This makes program management 

35 complex and is not practical. 

[01 50] In the second embodiment, such a signal for instructing to resend video data from the beginning is transmitted 
to the server as the contents distribution source using the " back-channel" ("back^hannel 1" or "back-channel 2 M 
described in the first embodiment, together with URL information of the request destination server and information 
representing the verification result. 

40 [0151] More specifically, in the second embodiment, from the player side which receives an MPEG-4 bit stream and 
plays back a scene (downstream processing) in normal use, information is distributed to the server side using the back- 
channel function of the MPEG-4 (i.e., verification/resending information is upstream-processed using Upchannel in- 
formation as shown in Fig. 1). In this method, each contents distribution source server and IPMP System Interface 
share sections associated with verification/resending information communication, so the cumbersomeness of program 

45 management can be decreased. 

[0152] As described above : according to the second embodiment, since copyright work resending request can be 
easily transmitted through the network after verification processing, video data to be played back can be prevented 
from being omitted due to the delay time associated with verification processing. 

[0153] In the second embodiment, the verification processing method is not particularly specified. More specifically, 
50 as in the first embodiment, a verification request signal is sent to each contents distribution server through the network 
to receive access permission from the contents distribution server. Alternatively, a valid key is stored in the MPEG-4 
player in advance, and the viewer locally makes verification. 

[0154] The present invention may be applied to a system constituted by a plurality of devices or an apparatus com- 
prising a single device. 

55 [01 55] In addition, an apparatus or method comprising some of all constituent elements of the apparatus or method 
of the above embodiments also constitutes the invention intended by the present inventor of this application. 
[0156] The function of the apparatus according to the above embodiments is realiied even by permanently or tem- 
porarily incorporating a storage medium storing program codes in a system or apparatus and causing the computer 
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(or a CPU or an MPU) of the system or apparatus to read out and execute the program codes stored in the storage 
medium. In this case, the program codes read out from the storage medium or the storage medium constitutes the 
invention by itself. 

[0157] As a storage medium for supplying the program codes, a floppy disk, a hard disk, an optical disk, a magne- 
5 tooptical disk, a CD-ROM : a CD-R. a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used, 
though another device may be used. 

[0158] Not only a case wherein the functions unique to the present invention are realized by executing the program 
codes read out from the storage medium by the computer but also embodiments in which an OS (Operating System) 
running on the computer performs part or all of actual processing on the basis of the instructions of the program codes 

10 also belongs to the technical range of the present invention. 

[0159] Embodiments in which after the program codes read out IrorrUhe storage medium are written in the memory 
of a function expansion board inserted into the computer or a function expansion unit connected to the computer, the 
CPU of the function expansion board or function expansion unit performs part or all of actual processing on the basis 
of the instructions of the program codes also belongs to the technical range of the present invention. 

is [0160] As has been described above, according to the present invention, a plurality of object streams and a stream 
associated with copyright information are transmitted as one stream, and the stream associated with copyright infor- 
mation is separated and extracted on the reception side. With this arrangement, verification processing can be efficiency 
executed to effectively protect copyrights and effectively use copyright works. 

[0161] In addition, according to the present invention, verification processing can be efficiently executed to effectively 
20 protect copyrights and effectively use copyright works. 

[0162] Furthermore, according to the present invention, after verification processing, a copyright- resending request 1 
is transmitted through the network. With this arrangement, omission of a played back image due to the delay time" 1 
associated with verification processing can be-prevented, and therefore, many verification processing methods become 

pOSSible. * £ ; .-TBCcPJ. 

25 [0163] As many apparently widely different; embodiments of .the present invention can be made without departing--* 
from the spirit and scope thereof, it is to be understood that the invention is not limited to the specific embodiments 
thereof except-as defined in the appended claims. . . . ..- ^ 

30 Claims 

1 . A verification apparatus comprising: 

reception means for receiving moving image data containing use permission information from an external.: 

35 information processing apparatus; 

playback means for playing back part of the moving image data received by said reception means; 

input means for inputting characters/symbols; * 

comparison means for comparing the use permission information contained in the moving image data received 

by said reception means with the characters/symbols input by said input means; and 

40 transmission means for, when comparison by said comparison means has revealed that the use permission 

information matches the characters/symbols, transmitting an instruction signal for instructing to play back the 
moving image data again from first image data played back by said playback means to said external information 
processing apparatus. 

45 2. The apparatus according to claim 1 , wherein the moving image data is compression-coded moving image data. 

3. The apparatus according to claim 1 , wherein said apparatus further comprises storage means for storing the char- 
acters/symbols in advance, and 

50 the comparison means compares the use permission information contained in the moving image data received 

by said reception means with the characters/symbols stored in said storage means. 

4. The apparatus according to claim 1 , wherein said playback means plays back time or spatial. part of the moving 
image data. 



55 



5. The apparatus according to claim 1 , wherein said transmission means transmits, together with the instruction 
signal, a signal representing that said verification apparatus can use the moving image data. 
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6. The apparatus according to claim 1 , wherein said transmission means normally receives the moving image data 
using an interface used to receive the moving image data in a normal direction and transmits the instruction signal 
using the interface in a reverse direction to the normal direction. 

5 7. The apparatus according to claim 1 , wherein said transmission means transmits the instruction signal using an 
upstream of an MPEG4 bit stream. 

8. The apparatus according to claim 1 , wherein said apparatus further comprises an interface connected to a com- 
munication line, and 

10 

said transmission means transmits the instruction signal through the interface and communication line. 

9. A verification apparatus comprising: 

is reception means for receiving moving image data constructed by a plurality of object data including object 

data containing source information and use permission information; 

separation means for separating the moving image data received by said reception means into the plurality 
of object data; 

playback means for playing back, of the object data separated by said separation means, object data which 
20 do not contain the use permission information; 

input means for inputting characters/symbols; 

comparison means for comparing the use permission information with the characters/symbols input by said 
input means in association with, of the object data separated by said separation means, the object data con- 
taining the use permission information; and 
25 tr«msmission means for, when comparison by said comparison means has revealed that the use permission 

mlormation matches the characters/symbols, transmitting an instruction signal for instructing to play back the 
moving image data again from first image data played back by said playback means to an external information 
processing apparatus corresponding to the source information contained in the plurality of object data con- 
st rjclmg the moving image data. 

30 

10. The apparatus according to claim 9, wherein 

the source information is URL (Uniform Resource Locator) information. 

35 11. A venfcation method comprising: 

the reception step of receiving moving image data containing use permission information from an external 
mlormation processing method; 

the playback step of playing back part of the moving image data received in the reception step; 
40 the input step of inputting characters/symbols; 

the comparison step of comparing the use permission information contained in the moving image data received 
in the reception step with the characters/symbols input in the input step; and 

the transmission step of, when comparison in the comparison step has revealed that the use permission in- 
formation matches the characters/symbols, transmitting an instruction signal for instructing to play back the 
45 moving image data again from "first image data played back in the playback step to said external information 

processing method. 

12. The method according to claim 11, wherein the moving image data is compress ion -coded moving image data. 

so 1 3. The method according to claim 11, wherein the method further comprises the storage step of storing the characters/ 
symbols in advance, and 

the comparison step comprises comparing the use permission information contained in the moving image data 
received in the reception step with the characters/symbols stored in the storage step. 

55 

14. The method according to claim 11, wherein the playback step comprises playing back time or spatial part of the 
moving image data. 
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15. The method according to claim 11 , wherein the transmission step comprises transmitting, together with the instruc- 
tion signal, a signal representing that the verification method can use the moving image data. 

16. The method according to claim 11 , wherein the transmission step comprises normally receiving the moving image 
data using an interlace used to receive the moving image data in a normal direction,and transmitting the instruction 
signal using the interface in a reverse direction to .the normal, direction. 

17. The method according to claim 11, wherein the transmission step comprises transmitting the instruction. signal 
using an upstream of an MPEG4 bit stream. 

18. The method according to claim 11 , wherein the method further comprises using an interface connected to a com- 
munication line, and 

the transmission step comprises transmitting the instruction signal through the interface and communication 

line. 

19. A verification method comprising: 

the reception step of receiving moving image data constructed by a plurality of object data including object 
data containing source information and use permission information; , 

the separation step of separating the moving image data received in the reception step into the plurality of* 
object data; * 
the playback step of playing back, of the object data separated in the separation step, object data which do 
not contain the use permission information; v 

the input step of inputting characters/symbols; mrtze- 
the comparison step of comparing the use permission information with the characters/symbols input in the 
input step in association with, of the object data separated in the separation step, the object data containing r 
the use permission information; and . , < ^ 

the iransmission step of, when comparison in the comparison step has revealed that the use permission in-": 
formal ion matches the characters/symbols, transmitting an instruction signal for instructing to play back the 
moving image data.again from first image data played back in the playback step to an external informations 
processing method corresponding to the source information contained in the plurality of object data construct- 
ing the moving image data.- . 

20. The method according to claim 19, wherein the source information is URL (Uniform Resource Locator) information. 

21 . A verification system having an information processing apparatus and a verification apparatus, which are connect- 
ed to communicate with each other, characterized in that said verification apparatus comprises: 

reception means for receiving moving image data containing use permission information from said information 
processing apparatus: 

playback means for playing back part of the moving image data received by said reception means; 
input means for inputting characters/symbols; . 

comparison means for comparing the use permission information contained in the moving image data received 
by said reception means with the characters/symbols input by said input means; and 
transmission means for, when comparison by said comparison means has revealed that the use permission 
information matches the characters/symbols, transmitting an instruction signal for instructing to play back the 
moving image data again from first image data played back by said playback means to said information 
processing apparatus. 

22. A verification system having at least one information processing apparatus and a verification apparatus connected 
to said information processing apparatus to communicate with each other, wherein said verification apparatus 
comprises: 

reception means for receiving moving image data constructed by a plurality of object data including object 
data containing source information and use permission information from said information processing appara- 
tus; 

separation means for separating the moving image data received by said reception means into the plurality 
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of object data; 

playback means for playing back, of the object data separated by said separation means, object data which 
do not contain the use permission information; 
input means for inputting characters/symbols: 
5 comparison means for comparing the use permission information with the characters/symbols input by said 

input means in association with, of the object data separated by said separation means, the object data con- 
taining the use permission information; and 

transmission means for, when comparison by said comparison means has revealed that the use permission 
information matches the characters/symbols, transmitting an instruction signal for instructing to play back the 
10 moving image data again from first image data played back by said playback means to the information process- 

ing apparatus corresponding to the source information contained in the plurality of object data constructing 
the moving image data. 

23. A storage medium which stores a program that can be executed by an apparatus, wherein said apparatus for 
75 executing the program is caused to operate as an apparatus comprising: 

reception means for receiving moving image data containing use permission information from an external 
information processing apparatus; 

playback means for playing back part of the moving image data received by said reception means; 

20 input means for inputting characters/symbols; 

comparison means for comparing the use permission information contained in the moving image data received 
by said reception means with the characters/symbols input by said input means; and 
transmission means for, when comparison by said comparison means has revealed that the use permission 
information matches the characters/symbols, transmitting an instruction signal for instructing to play back the 

2S moving image data again from first image data played back by said playback means to said external information 

processing apparatus. 

24. A storage medium which stores a program that can be executed by an apparatus, wherein said apparatus for 
executing the program is caused to operate as an apparatus comprising: 

30 

reception means for receiving moving image data constructed by a plurality of object data including object 
data containing source information and use permission information; 

separation means for separating the moving image data received by said reception means into the plurality 
of object data; 

35 playback means for playing back, of the object data separated by said separation means, object data which 

do not contain the use permission information; 
input means for inputting characters/symbols; 

comparison means for comparing the use permission information with the characters/symbols input by said 
input means in association with, of the object data separated by said separation means, the object data con- 

40 taining the use permission information; and 

transmission means for, when comparison by said comparison means has revealed that the use permission 
information matches the characters/symbols, transmitting an instruction signal for instructing to play back the 
moving image data again from first image data played back by said playback means to an external information 
processing apparatus corresponding to the source information contained in the plurality of object data con- 

45 structing the moving image data. 



25. A verification apparatus comprising: 



separation means for separating image data constructed by a plurality of object data containing source infor- 
50 mation into the plurality of object data and source information of the plurality of object data; 

management means for managing the plurality of source information separated by said separation means; 
transmission means for transmitting a verification signal to an information device on a network, which is spec- 
ified by arbitrary source information managed by said management means; 

reception means for receiving a permission signal transmitted from said information device in response to the 
55 verification signal transmitted by said transmission means; and 

control means for enabling use of object data containing the arbitrary source information on the basis of the 
permission signal received by said reception means. 
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26. The apparatus according to claim 25, wherein the source information is URL (Uniform Resource Locator) infor- 
mation. 



The apparatus according to claim 25, wherein the image data constructed by the. plurality of object data is com- 
pression-coded moving image data. 

The apparatus according to claim 25, further comprising 

selection means for selecting arbitrary source information managed by said management means. 

The apparatus according to claim 25, wherein said transmission. means normally receives the image data using 
an interface used to receive the- image data constructed by the plurality of object data in a normal direction and 
transmits the verification signal using the interface in a reverse direction to the normal direction. 

is 30. The apparatus according to claim.25, wherein said transmission means transmits the verification signal using an 
upstream of an MPEG4 bit stream. 

31. The apparatus according to claim 25, wherein said apparatus further comprises an interface connected to a com- 
munication line, 

20 

said transmission means transmits the verification signal to said information device through the interface andv 

communication line, and * 

said reception means receives the permission signal from said information device through the communication 
line and interface. idtenr 

2S 

32. A verification method comprising: 



the separation step of separating image data constructed by a plurality of object data containing source infor- l 

mation into the plurality of object data and source information of the plurality of object data; 
30 the management step of managing the plurality. of source?information separated in the separation. step; 

the transmission step of transmitting a verification signal to an information device on a network, which is spec-r. 

ified.by arbitrary source information managed in the management step; = ■•- * > ■ 

the reception step of receiving a permission signal transmitted from said information device in response to the 

verification signal transmitted in the transmission step; and 
35 the control step of enabling use of object data containing the arbitrary source information on the basis of the 

permissionsignal received in the reception step. . • ,, 

33. A verification system having an information device and a verification apparatus, which are. connected to a network, 
wherein said verification apparatus comprises: 

40 

separation means for separating image data constructed by a plurality of object data containing source infor- 
mation into the plurality of object data and source information of the plurality of object data; 
management means for managing the plurality of source information separated by said separation means; 
transmission means for transmitting a verification signal to said information device on a network, which is 
45 specified by arbitrary source information managed by said management means; 

reception means for receiving a permission signal.transmitted from said information device in response to the 
verification signal transmitted by said transmission means; and 

control means for enabling use of object.data containing the arbitrary source information on the basis of the 
permission signal received by said reception means. 

so 

34. A storage medium which stores a program for controlling verification processing, wherein an apparatus for loading 
and executing the program is caused to operate as an apparatus comprising: 

separation means for separating image data constructed by a plurality of object data containing source infor- 
55 mation into the plurality of object data and source information ol the plurality of object data; 

management means for managing the plurality of source information separated by said separation means; 
transmission means for transmitting a verification signal to an information device on a network, which is spec- 
ified by arbitrary source information managed by said management means; 
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reception means for receiving a permission signal transmitted from said information device in response to the 
verification signal transmitted by said transmission means; and 

control means for enabling use of object data containing the arbitrary source information on the basis of the 
permission signal received by said reception means. 

5 

35. A verification apparatus comprising: 

coding means for coding streams of object data constructing image data and streams of source information 
of the object data into one stream; 
w generation means for generating a decoded stream from the plurality of streams coded by said coding means; 

separation means for separating the decoded stream generated by said generation means into the streams 
of the plurality of object data contained in the decoded stream and the streams of the source information of 
the plurality of object data; 

management means for managing the source information of the object data; and 
75 transmission means for transmitting the plurality of streams of source information separated by said separation 

means to said management means. 

36. The apparatus according to claim 35, wherein the source information is URL (Uniform Resource Locator) infor- 
mation. 

20 

37. The apparatus according to claim 35, wherein the image data constructed by the plurality of object data is com- 
pression-coded moving image data. 

38. The apparatus according to claim 35, further comprising 

25 

designation means for designating a type of each stream coded by said coding means. 



39. The apparatus according to claim 38, further comprising 

30 identification means for identifying the type of each stream separated by said separation means, and 

sending means for sending each stream to an appropriate decoder on the basis of the type identified by said 
identification means. 



3S 



45 



so 



40. A verification method comprising: 



the coding step of coding streams of object data constructing image data and streams of source information 
of the object data into one stream; 

the generation step of generating a decoded stream from the plurality of streams coded in the coding step; 
the separation step of separating the decoded stream generated in the generation step into the streams of 
40 the plurality of object data contained in the decoded stream and the streams of the source information of the 

plurality of object data; 

the management step of managing the source information of the object data; and 

the transmission step of transmitting the plurality of streams of source information separated in the separation 
step to the management step. 



41. The method accordingto claim 40, wherein the source information is URL (Unilorm Resource Locator) information. 

42. The method according to claim 40, wherein the image data constructed by the plurality of object data is compres- 
sion-coded moving image data. 

43. The method according to claim 40, further comprising 

the designation step of designating a type of each stream coded in the coding step. 



ss 44. The method according to claim 43, further comprising: 



the identification step of identifying the type of each stream separated in the separation step, and 

the sending step of sending each stream to an appropriate decoder on the basis of the type identified in the 
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identification step. 

45. A verification system comprising a transmission apparatus and a reception apparatus, wherein said transmission 
apparatus comprises: 

5 

coding means for coding streams of object data constructing image data and streams of source information 
of the object data into one stream; 

generation means for generating a decoded stream from the plurality of streams coded by said coding means; 
and 

10 transmission means for transmitting the decoded stream generated by said generation means to said reception 

apparatus, and 

said reception apparatus comprises: 

is separation means for separating the decoded stream transmitted from said transmission apparatus into the 

streams of the plurality of object data contained in the decoded stream and the streams of the source infor- 
mation of the plurality of object data; 

management means for managing the source information of the object data; and 

sending means for sending the plurality of streams of source information separated by said separation means 
20 to said" management means. ?; • 

46. A storage medium which stores a. program that can be executed by. an apparatus, wherein. said, apparatus for 
executing the program is caused to operate as an apparatus comprising: . . . . 

25 coding means for coding streams of object data constructing image data and streams of source information 

of the object data into one stream; . ^ 

generation means for generating a decoded stream from the plurality of streams coded by said coding means; 
separation means for separating the decoded stream generated by said generation means into the streams: 
of the plurality of object data contained in the decoded stream and the streams of the source information of 

30 the plurality of object data; 

management means fo> managing the source information of the object data: and 

transmission means for transmitting the plurality of streams of source information separated by said separation 
means to said management means. 

35 47. A method of transmitting video image data in an MPEG4 data stream in^vyhichjjse^ermission data is encoded 
with the video image data and, to avoid loss of image data in periods during which the use-permission data is 
processed at a receiver to establish use-permission, the image data for at least such periods is retransmitted. 

48. A computer program comprising processor implementable instructions for controlling a receiving apparatus to carry. 
40 out a method as claimed in any of claims 11 to 20, 32 and 40 to 44. 



45 



SO 



SS 



EP 0 969 668 A2 



FIG. 1 



NETWORK 



30 



TCP/IP 



1 



10 



DIGITAL VIDEO DATA 
DISTRIBUTION SERVER 



CONVERSION SECTION 



11 



1 



12 



DIGITAL VIDEO 
DATA STORAGE 
DEVICE 



20 ,22 



TCP/IP 


DIGITAL VIDEO DATA 
RECEPTION CLIENT 




DIGITAL VIDEO 
DATA STORAGE 
DEVICE 






CONVERSION SECTION 




-21 





EP 0 969 668 A2 




TJ CP - 

' Interactive 
Audiovisual 
--"Scene 




FIG. 2 



Display And 
User Interaction 



Composition and Rendering 



Compression 
Layer 

_j 



Object 
Descriptor 




Scene 1 
Description 
Information 



Primitive 
AV Objects 



Upchannel 
Information 



Elementary 
Stream 



Elementary 

Stream 

Interlace 



Sync 
fSLl Layer 



SL-Packetized Streams 



Stream Multiplex 
Interface 



FlexMux 



FlexMux 



FlexMux 



TransMux 
Layer 



(PES) 


(RTP) 


AAL2 


H223 


DAB 




MPEG-2 


UDP 


ATM 


PSTN 


Mux 




TS 


IP 











V 



TransMuxStreams 



Transmission/Storage Medium 



EP 0 969 668 A2 



CO 

d 

LL 









» — 




o 




g. 


CD 




O 




CO 








Q 



E 

03 

175 





EP 0 969 668 A2 



00 

o 

C\J 





CO ! 






CD 


o 


C 




CD 




O 


o 


CO 


1 Des 




EP 0 969 668 A2 



FIG. 5 




Content Address 




Initial 
Object 
Desriptor 



ES .descriptor (optional) 
for ObjectDescriptorStream 



ESjjescriptor 

for SceneDescriptionStream 



ES_descriptor (optional) 
for IPMPStream 
or ObjectDescritorStream 
for IPMPSteam 



handle for 

ObjectDescriporStream 
handle for 

ObjectDescriptionStream 

handle for 
IPMPStream 



EP 0 969 668 A2 



FIG. 6 



Initial Object 
Descriptor 



! BIFS Update (Replace Scene) 



ES Descr 



ES Descr 



-Scene Description 
-Stream 



ESJD 




ObjectDescriptorUpdate 



_ Object Descriptor ID 



Object Descriptor \^ 
Stream 



Initial 
Object 
Descriptor 
URL 



Object 
Descriptor^ 




Visual 
Stream 



Audio 
\ Stream 



Initial Object 
Descriptor 



BIFS Update (Replace Scene) 




Object Descriptor ID 



EP 0 969 668 A2 



FIG. 7 

( START ) 

RECEIVE MOVING 
IMAGE DATA 



SEPARATE MOVING 
IMAGE DATA 




SELECT URL 
INFORMATION 



TRANSMIT VERIFICATION 

REQUEST SIGNAL TO 
CORRESPONDING SERVER 



YES 



S6 
_1 



S5 

HAS ACCESS' 
PERMISSION SIGNAL BEEN 
RECEIVED FROM 
SERVER?. 



PERMIT ACCESS 
TO OBJECT 



S3 



S4 



S7 



INHIBIT ACCESS 
TO OBJECT 




EP 0 969 668 A2 



FIG. 8 




Composition Layer 



Compression Layer 
k 



a a a 



IPMP Stream Flow 
Control 



a a ';, ' a A A 



a a -A ' a '. A 



Demux Layer 

~" A 



MPEG-4 Bit Stream 







Sync Layer 






IPMP-Elementary Streams 
IPMP-Descriptors 



EP 0 969 668 A2 




EP 0 969 668 A2 



FIG. 10 



MPEG-4 stream 



I 



91 



FlexDemux 



92 



ALManager 



RECEIVE MPEG-4 BIT STREAM AND . . 
SEPARATE IT INTO ES IN UNITS OF 
OBJECTS 

DIVIDE ES OF EACH OBJECT IN ..UNITS 
OF DECODING UNITS 



1 



93,94 



BIFS Decoders 
Decoders 



DECODE EACH OBJECT 




DATA GROUP TO BE DECODED IN 
UNITS OF OBJECTS ' 



ADJUST TIME OF EACH OBJECT, 
SYNTHESIZE SCENE, AND DISPLAY IT 



MediaStreamlmp "Fetch 0 



EP 0 969 668 A2 



FIG. 11 



ADD ALLOWANCE VALUE 
TO CURRENT TIME 
-»dwCurrentTime 



OBTAIN CURRENT 
DECODING UNIT(AU) 



OBTAIN TIMESTAMP 
{STAMP TIME) OF AU 
— dwTime 



S1104 



TO NEXT PROCESSING 
(SCENE SYNTHESIS) 



S1101 



S1102 



S1103 




S1105 



OBTAIN NEXT DECODING 
UNIT (AU) 



S1106 



EP 0 969 668 A2 




EP 0 969 668 A2 



FIG. 13 



MPEG-4 


stream 

, ,91 


FlexDemux 




, /92 


ALManager 



RECEIVE MPEG-4 BIT STREAM AND 
SEPARATE IT INTO ES IN UNITS OF 
OBJECTS 

DIVIDE ES OF EACH OBJECT IN UNITS 
OF DECODING UNITS 




SPECIFY PROTECTED STREAM AND 
INTERPRET IT USING VALID "KEY" 



DATA GROUP TO BE DECODED IN 
UNITS OF OBJECTS 



93,94 



BIFS Decoders 
Decoders 



L 



95 



Presenter 



DECODE EACH OBJECT 



ADJUST TIME OF EACH OBJECT, 
SYNTHESIZE SCENE, AND DISPLAY IT 



EP 0 969 668 A2 



FIG. 14 



YES 



OBTAIN STREAM OF 

EACH OBJECT 
(ALManeger, SyncLayer) 



S1401 



S1402 




HOLD WITHOUT 
INTERPRETING PROTECTED 
STREAM 



-S1403 



INTERPRET PROTECTED 
STREAM AND ADVANCE TO 
NEXT PROCESSING 



S1404 



(19) 



Europaisches Patentamt 
European Patent Office 
Office europeen des brevets 




(12) 



(88) Date of publication A3: 

13.03.2002 Bulletin 2002/11 

(43) Date of publication A2: ' 

05.01.2000 Bulletin 2000/01 

(21 ) Application number: 99305062.4 

(22) Date of filing: 28.06.1999 



(H) EP 0 969 668 A3 

EUROPEAN -PATENT APPLICATION 

(51) mt ci 7: H04N 7/24, G06F 1/00 



(84) Designated Contracting States: 


(72) Inventors: 


AT BE CH CY DE DK ES Fl FR GB GR IE IT LI LU 


• lnoue,'Hiroshi 


MC NLPT SE 


Ohta-ku, Tokyo (JP) 


Designated Extension States: 


• Nakagawa, Toshiyuki 


AL LT LV MK RO SI 


Ohta-ku, Tokyo (J P) 


(30) Priority: 29.06.1998 JP 18303498 


(74) Representative: , . 


16.10.1998 JP 29593698 


Beresford, Keith Denis Lewis et al 


16.10.1998 JP 29593798 


BERESFORD & Co. 




2-5 Warwick Court, . 


(71) Applicant: CANON KABUSHIKI KAISHA 


HighHolborn - , 


Tokyo (JP) 


London WC1R5DH(GB) 



CO 

< 

00 
CO 

co 

CO 



Q. 
LU 



(54) Copyright protection for moving image data 

(57) In a system for playing back a moving image 
constructed by a set of objects, each object is eifficiency 
verified, and a problem of omission of a played back im- 
age due to delay time associated with verification 
processing is solved. On the transmission side, an MUX 
(201 ) generates an MPEG-4 bit stream obtained by .cod- 
ing a plurality of object and data streams constructing a 
moving image and IPMP (Intellectual Property Manage- 
ment and Protection) streams containing source infor- 
mation and use permission information into one stream. 
On the reception side, a DMUX (206) identifies and sep- 
arates each stream from the MPEG-4 bit stream and 
separates them into a plurality of object data and source 
information/use permission information corresponding 



to each object data. On the basis of the sepa rated 
source information/use permission information, an 
IPMP System (207) requests a corresponding server 
(202 - 204) to execute verification to access each object 
using a back-channel 1 (205) using an upstream or 
back-channel 2 (213) such as a telephone line. An ac- 
cess control point (212) is controlled to permit only ac- 
cess to object data for which an access permission sig- 
nal has been received from the server. A signal for re- 
questing resending objects is transmitted to the distri- 
bution server (202 - 204) of objects for which verification 
is requested, using the back-channel 1 (205) or 2 (213), 
together with the source information of the request des- 
tination server and information representing the verifi- 
cation result. 



FIG. 4 




EP 0 969 668 A3 



J 



European Patent 
Office 



EUROPEAN SEARCH REPORT 



Application Number 

EP 99 30 5062 



DOCUMENTS CONSIDERED TO BE RELEVANT 



Category 



Citation of document with indication, where appropriate, 
of relevant passages 



Relevant 
to ciaim 



CLASSIFICATION OF THE 
APPLICATION (trrt.CI.7) 



US 5 003 384 A (DURDEN GREGORY S ET AL) 
26 March 1991 (1991-03-26) 

* column 2, line 56 - column 3, line 2 * 

* column 6, line 30 - line 61 * 

* column 7, line 14 - line 40 * 

* figure 1 * 

US 5 619 249 A (MARTIN ROBERT I M ET AL) 
8 April 1997 (1997-04-08) 

* column 3, line 9 - line 13 * 

* column 4, line 26 - line 32 * 

* column 13, line 33 - line 43 * 



WO 97 14087 A (ERICKSON JOHN S) 
17 April 1997 (1997-04-17) 
* abstract * 
page 6, line 29 - page 7, line 13 * 
page 8, line 1 - line 6 * 
page 8, line 26 - page 9, line 23 * 
page 11, line 1 - line 17 * 
page 19, line 29 - page 20, line 8 * 
page 21, line 23 - page 22, line 4 * 
page 39, line 19 - line 25 * 
figures 4,5 * 



1-24,47, 
48 



H04N7/24 
G06F1/00 



1-24,47, 
48 



25-46 



TECHNICAL FIELDS 
SEARCHED (lnt.CJ.7) 



H04N 
G06F 



-/- 



The present search report has been drawn up tor all claims 



THE HAGUE 



Oats ol compaction of me search 

9 January 2002 



Examiner 

Fantini, F 



CATEGORY OF CITED DOCUMENTS 

X : pan leu tarty relevant It taken alone 

Y : particularly relevant If combined with another 

document of the same category 
A : tecrinotogJcal background 
O : non-wrtten dtectosure 
P : imermedJace document 



T : theory or principle underlying the invention 
E : eerier patent document, but pu Wished on, or 

after the OiinQ date 
D : document cited in the application 
L : document cited for other reasons 



& : member ol the same patent tamfly. corresponding 
document 



EP 0 969 668 A3 




European Patent 
Office 



EP 99 30 5062 



Application Number 



CLAIMS INCURRING FEES 



The present'European patent application compreed at the time of filing more than ten claims. 

□ Only part of the claims have been paid within the prescribed time limit. The present European search 
report has been drawn up for the first ten claims and for those claims for which claims lees have > 
been paid, namely claim(s): : 



□ No claims lees have been paid within the prescribed time limit. The present European search report has 
been drawn up for the first len claims. 



LACK OF UNITY OF INVENTION 



The Search Division considers that the present European patent application does not comply with the 
requiremenls of unity of invention and relates to several inventions or groups of inventions, namely; 



see sheet B 



□ All further search fees have been paid within the fixed time limit. The present European search report has 
been drawn up for all claims. 

F] As all searchable claims couid be searched without effort justifying an additional fee, the Search Division 
I— I did not invite payment ol any additional fee. - '. « * \ 

□ Only part of the further search fees have been paid within the fixed time limit The present European 
search report has been drawn up for those parts of the European patent application which relate to the 
inventions in respect of which search fees have been paid, namely claims: 



I I None ol the further search fees have been paid within the lixed time limit. The present European search 
I — I report has been drawn up lor those parts of the European patent application which relate to the invention 
first mentioned in the claims, namely claims: 



EP 0 969 668 A3 



European Patent 
Office 



EUROPEAN SEARCH REPORT 



Application Number 

EP 99 30 5062 



DOCUMENTS CONSIDERED TO BE RELEVANT 



Category 



Citation of document with indication, where appropriate. 
of relevant passages 



Relevant 
to claim 



CLASSIFICATION OF THE 
APPLICATION <hIt.CI.7) 



"INTERNATIONAL ORGANIZATION FOR 
STANDARDIZATION - ORGANISATION 
INTERNATIONALE DE NORMAL ISAT ION, XX , XX" 
INTERNATIONAL ORGANIZATION FOR 
STANDARDIZATION - ORGANISATION 
INTERNATIONALE DE NORMALISATION, XX, XX, 

15 May 1998 (1998-05-15), XP002133316 



1,7, 

9-11,17, 
19-48 



paragraph 
paragraph 
paragraph 
paragraph 
paragraph 
paragraph 
paragraph 
paragraph 



'1.6.1 
'1.6.5 
'8.3 
'8.3 
'8.3 
'8.3 
'8.3.8 
'8.4.3 



ISO/TC 46/SC 9/WORKING GROUP 1: "Managing 

Intellectual Property Identification and 

Protection within HPEG-4" 

ISO PROJECT 15706: INTERNATIONAL STANDARD 

AUDIOVISUAL NUMBER (ISAN), 'Online! 

XP002171214 

Retrieved from the Internet: 
<URL : http : //searchpdf . adobe . com> 
'retrieved on 2001-07-04! 
* the whole document * 



1,7, 

9-11,17, 
19-48 



TECHNICAL FIELDS 
SEARCHED (lntCI.7) 



US 5 282 249 A (COHEN MICHAEL 
25 January 1994 (1994-01-25) 



* abstract * 

* figures 1.1,1.2,5,6 * 



ET AL) 



1,9,11, 
19, 

21-24, 
47,48 



The present search report has been drawn up for ati claims 



Place ot taarch 

THE HAGUE 



Data of oompWton ot the seercti 

9 January 2002 



Fantini, F 



CATEGORY OF CITED DOCUMENTS 

X : particuJarfy relevant If taken alone 

v : particularly relevant rt combined with another 

document of the sane category 
A : technological background 
O : non— written disclosure 
P : intermediate document 



T : theory or principle underlying the Invention 
E : earlier patent document, but published on. or 

after the fling date 
D : document cited In the application 
L : document eked for other reasons 

& : member of the same patent family, corresponding 
document 



EP 0 969 668 A3 




European Patent 
Office 



LACK OF UNITY OF INVENTION 
SHEET B 



EP 99 30 5062 



Application Number 



The Search Division considers that the present European patent application does not comply with the 
requirement of unity of invention.and relates to several inventions or groups of inventions, namely: 

1. Claims: 1-24, 47-48 

A verification involving the transmission of an instruction 
signal for instructing to play back moving image data again 
from first image data. 



2. Claims: 25-46 



A verification involving transmission of a verification 
signal to an information device on a network, reception 
permission signal enabling use of object data. 



EP 0 969 668 A3 



European Patent 
Office 



EUROPEAN SEARCH REPORT 



Application Number 

EP 99 30 5062 



DOCUMENTS CONSIDERED TO BE RELEVANT 



Category 



Citation of document with indication, where appropriate, 
of relevant passages 



Relevant 
to claim 



CLASSIFICATION OF THE 
APWJCATION (lntCI.7) 



ne 5 * 



US 5 619 501 A (CHANEY JOHN W ET AL) 
8 April 1997 (1997-04-08) 



* abstract * 

* column 4, line 36 - column 5, line 3 * 

* figure 3 * 

WO 97 23997 A (WEBBER ALUN DAVID ;CR0SSLEY 
ROBIN (GB); HOLLIDAY DAVID (GB); JAMES) 
3 July 1997 (1997-07-03) 

* page 9, line 5 - line 10 * 

* page 13, line 6 - line 12 * 

* page 29, line 16 - page 31, li 

* figures 2,11 * 

EP 0 778 513 A (MATSUSHITA ELECTRIC IND CO 
LTD) 11 June 1997 (1997-06-11) 

* abstract * 

* column 2, line 43 - column 3, line 5 * 

* column 3, line 54 - column 5, line 3 * 

* column 5, line 47 - column 6, line 51 * 

* column 10, line 29 - line 50 * 

* figure 1 * 

EP 0 715 246 A (XEROX CORP) 
5 June 1996 (1996-06-05) 

* abstract * 

* page 3, line 54 - page 4, line 12 * 

* page 5, line 2 - line 9 * 

* page 15, line 28 - line 39 * 

* page 21, line 52 - page 22, line 12 * 

* figures 1,4B * 



The present search report has been drawn up for all claims 



1,9,11, 
19, 

21-24, 
47,48 



1,9,11, 
19, 

21-24, 
47,48 



25-46 



25-46 



TECHNICAL FIELDS 
SEARCHED (lnl.CI.7) 



PIsc* d March 



THE HAGUE 



Date ol comptstion ol the search 

9 January 2002 



Examiner 

Fantini , F 



CATEGORY OF CITED DOCUMENTS 

X particularly ret&vant if taken atone 
- Y . particularly relevant if combined with another 

document of the Game category 
A . tecnnoiogical background 
O ' rcoo— written disclosure 
P ■ tntermottate document 



T : theory or principle underlying the invention 
E : earlier patent document, but published on, or 

after the filing date 
D : document died tn the application 
L : document cited for other reasons 

& : member of the same patent family, corresponding 
document 



EP 0 969 668 A3 



ANNEX TO THE EUROPEAN SEARCH REPORT 

ON EUROPEAN PATENT APPLICATION NO. EP 99 30 5062 



This annex lists the patent family members relating to the patent documents cited in the above-mentioned European search report 
The members are as contained in the European Patent Office EDP file on 

The European Patent Office Is In no way liable for these particulars which are merely given for the purpose of information. 

09-01-2002 



Patent document 




Publication 






Patent family 


Publication 






date 








date 


US 5003384 


A 


26-03-1991 


AU 


3340989 A 


16-10-1989 








CA 


1337215 Al 


03-10-1995 








CN 


1037249 A 


15-11-1989 








EP 


0410987 Al 


06-02-1991 








JP 


4500889 X 


,13-02-1992 








MX 


168475 B 


26-05-1993 


• ■ " 






WO 


; 8909528 Al 


05-10-1989 


US 5619249 


;A 


- : :™ 

08-04-1997 r 


AU 


685320 B2 


15-01-1998 






. » ' '. 


AU 


3551695 A 


29-03-1996 


t 






XA 


2199153 Al 


.21-03-1996 


-r-. - - > A .. 






DE 


69506571 Dl 


21-01-1999 








:de 


69506571 T2 


12-05-1999 








DK 


781489 T3 


16-08-1999 








EP 


0781489 Al 


02-07-1997 


■ > ■ 






ES 


2126317 T3 


16-03-1999 








JP 


10505976 T 


09-06-1998 




r 




WO 


9608927 Al 


21-03-1996 


•■ 

. ■. ■ * 






US 


6314575 Bl 


06-11-2001 


WO 9714087 


A 


17-04-1997 


us 


5765152 A 


09-06-1998 


.>..■* 






AU 


; 7662496 A 


30-04-1997 








WO 


9714087 Al 


17-04-1997 


US 5282249 


A 


25-01-1994 


IL 


92310 A 


30-05-1994 








US 


5481609 A 


02-01-1996 








AT 


160662 T 


15-12-1997 








AU 


642157 B2 


14-10-1993 








AU 


6230690 A 


23-05-1991 








Pa 


2025585 Al 


15-05-1991 








DE 


69031757 Dl 


08-01-1998 








DE 


69031757 T2 


19-03-1998 








EP 


0428252 A2 


22-05-1991 








ES 


2111531 T3 


16-03-1998 








HK 


1005354 Al 


31-12-1998 








JP 


3199119 B2 


13-08-2001 








JP 


3210843 A 


13-09-1991 


US 5619501 


A 


08-04-1997 


CA 


2188127 Al 


02-11-1995 








CN 


1151233 A 


04-06-1997 








CN 


1167405 A 


10-12-1997 








DE 


69505369 Dl 


19-11-1998 








DE 


69505369 T2 


15-04-1999 








DE 


69523183 Dl 


15-11-2001 








EP 


0756801 Al 


05-02-1997 








EP 


0858222 Al 


12-08-1998 



For more details about this annex : see Official Journal of the'European Patent Office, No. 12/82 * 



EP 0 969 668 A3 



ANNEX TO THE EUROPEAN SEARCH REPORT 

ON EUROPEAN PATENT APPLICATION NO. EP 99 30 5062 



This annex lists the patent family membersrelating to the patent documents cited in the above-mentioned European search report. 
The members are as contained In the European Patent Office EDP file on 

The European Patent Office is in no way Gable for these particulars which are merely given tor the purpose of information. 

09-01-2002 



Patent document 


Publication 


Patent family 


Publication 


cited in search report 


date 


members) 


date 



US 5619501 A ES 2123243 T3 01-01-1999 

JP 9512675 T 16-12-1997 

W0 9529560 Al 02-11-1995 

US 5802063 A 01-09-1998 



W0 9723997 A 


03-07- 


•1997 


AU 


7631696 A 


17-07-1997 








CA 


2241368 Al 


03-07-1997 








EP 


1158802 A2 


28-11-2001 








EP 


0868816 Al 


07-10-1998 








WO 


9723997 Al 


03-07-1997 








JP 


2000502846 T 


07-03-2000 








NO 


982805 A 


20-08-1998 


EP 0778513 A 


11-06- 


-1997 


JP 


9160899 A 


20-06-1997 








CN 


1158456 A 


03-09-1997 








EP 


0778513 A2 


11-06-1997 








KR 


201550 Bl 


15-06-1999 








US 


5842023 A 


24-11-1998 


EP 0715246 A 


05-06- 


-1996 


US 


5638443 A 


10-06-1997 








EP 


0715246 Al 


05-06-1996 








JP 


8263439 A 


11-10-1996 



For more details about Ms annex : see Official Journal of the European Patent Office. No. 12/82 



This Page is Inserted by IFW Indexing and Scanning 
Operations and is not part of the Official Record. 

BEST AVAILABLE IMAGES 

Defective images within this document are accurate representations of the original 
documents submitted by the applicant. 

Defects in the images include but are not limited to the items checked: 

□ BLACK BORDERS 

□ IMA£E CUT OFF AT TOP, BOTTOM OR SIDES 
Ofadep TEXT OR DRAWING 

□Tblurred or illegible text or drawing 

□ skewed/slanted images 

□ color or black and white photographs 

□ grayscale documents 

olines or marks on original document 

□ reference(s) or exhibit(s) submitted are poor quality 

□ OTHER: 

IMAGES ARE BEST AVAILABLE COPY. 
As rescanning these documents will not correct the image 
problems checked, please do not report these problems to 
the IFW Image Problem Mailbox. 



